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POLICY RESEARCHER AS SLEUTK: 
APPRDCHES TO INTEGRATING QUALITATIVE AND QUANTITATIVE METHODS 



For Che past thirty years, ic has been common to refer to the exist- 
ance of two distinctive -paradigms" governing the methods of social science 
«nqui'ry. The first paradigm stresses the need to apply research design and 
analysis principles derived from the "hard" sciences, and emphasizes the 
desireabiUty of experimental or quasi-experimental design, and statistical 
analysis. A second paradigm argues that social phenomena are essentially 
different from those observed by the hard sciences and that, m order to 
urierstand them adequately, we must understand the ways in which they occur 
naturally, and their meaning to members of the social structure. A "holistic- 
understanding of human social structures and behaviors requires a qualitative, 
obsecvationally based methodology rather than experimental manipulation and 
analysis of a selected number of variables. As recently as 1977, one observer 
of these two camps commented that the gulf between them was so areat that it 
was unrealistic to assume th.t there would be any "grand synthesis" m 
the forseeable future, and that any steps toward synthesis were on the 
"fringes" of paradigms. 

However, there are a number of indications that a felt need for 
something more than simple detente between the camps was growing. Some 
experimental methodolog is»s , for example, have recently taken tentative 
steps toward not only acknowledging the existance of an alternative paradiam. 
but also Its suitaoility for studying phenomenon which have typically been 
dominated by quantitative a^.r .aches (Campbell, 1974; Cook and Cook, 1977). _ 
Similarly, researchers wno are advocates of qualitative methods have also 
called for greater attention to standardization of analysis procedures 
(Sieber, n.d.). finally, a number of key articles and books have advocated 
for integration between qualitative and quantitative methods within the same 
study (Sieoer, 1975; Lazarsfeld, 1976; Cook and Reichardt, 1979). The 
expressed need for integration is not occurring or ly at the fnnqe of social 
sciences disciplines, but is also supfx^rted at the center, and is becomna 
xore widespread. The tidiness of trhe divisions between camps is clearly 
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breaking down: one can no longer assume tl>at an Anthropologist is totally 
ignorant of statistics, and traditional experimental psychologists are 
hotly discussing the problems of small N designs for which preferred statis- 
tical techniques are in appropriate (Herson and Barlow, 1976; KrachtowUl, 
1977) . 

The movement toward integration between qualitative and quantita- 
tive methods has been fostered most evidently in social policy research for 
several reasons. First, some of the early high aspirations for quantita- 
tive social policy research were deflated by an accumulation of "null" find- 
ings, and -black box" research designs were unable to reveal why apparently 
massive experimental treatments should produce no measureable effects. Thus, 
committed empiricists began to look at qualitative research methods as an 
approach that might help them to improve their analysis - to point to inter- 
action effects that should be explored, to allow them to account for other- 
wise inexplicable findings, and so forth.* Other researchers, such as Gross, 
et al., (1972) or Charters and Pellegrin (1972), raised the need to examine 
whether or not a treatment had actually been implemented as part of an elabo- 
rate experimental or quasi-experimental design. These studies also indi- 
cated the difficulty of deteruuning the degree of unplementation without some 
qualitat ve understanding of what constituted implementation for a given pro- 
gram. 

second, the most rigorous and sophisticated of designs has not elim- 
inated doubts about the durability of policy research findings. Rather than 
eliminating controversy over the results o£ social policy and evaluation 
researcn, rigorous designs have simply raised a new dimension to the debate. 
Any observer who does not like the results of a major policy study can^almost 
invariably find a variety of methodological or analytic flaws which undermine 
its validity. Not surprisingly, some policy makers have arrived at a deep 
seated skepticism about the durability of supposedly "hard" findings - at 

*See Sie b^r (1975) for an extensive discussion of the ways in which 

qualitative research can be used to complement a Jesign which is predion nantly 
quantitative. 
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least whete they are unsupported by qualitative data which makes sense in 
the Mght of ordinary knowledge ar.d experience (Corbett and Firestone, 1980; 

Sundquist, 1978).* 

Third, there are also pratical considerations which have promoted 
the use of qualitative methods. The increasing burden of "forms clearance- 
procedures required before standardized data collection instruments can be 
used under federal contract regulations should not be underestimated as a 
motivator, both for federal agencies and researchers. Since forms clearance 
can take from four to six months, the federal agency that asks for qualita- 
tive data in addition to quantitative can begin to "know" something about 
the topic in question long before a survey or testing program could begin. 
Thus, particularly in cases _where there is only liirited interest in a "bot- 
tom line" assessment, qualitative approaches may be perceived as more effi- 
cient.** Qualitative designs may also be viewed as more flexible at respond- 
ing to changing policy coontexts and questions than traditional experimental 
designs. 

It should be emphasized that the ,above named pressures do not provide 
simply a shift in emphasis from one camp to another, but rather a desire on 
the part of policy makers (and at least some researchers) to draw upon the best 
of both methods. Despite the increase in policy makers' support of qualitative 
research there continues to be limited interest in sponsoring true ethnographic 
case studies, except in tht context of supplementing very large, well defined, 
social experiments (see, for example. Trend, 1976; Herriott, 1980). The new 
emphasis upon qualitative methods does not seeK a paradigm shift, but would 
retain the strengths of quantitative research - general izeability of results, 
reliability of obs*ervations , and the ability to synthesis a large complex 
study in a brief report. Increasingly, there has also been a strong perceived 
need to address the integration of findings across different methodological 



*That poli ' c y makers have come to view the "soft" approaches as fruitful 

13 evidenced by the significant increase in the number of RFPs fro. a variety 
of agencies which require qualitative or case based approaches rather than 
(or in addition to) quantitative oner.. 

*-That field based methods should coir.e to be thought of as efficient is 
an ironic turnabout from earlier periods in which "public opinion" surveys and 
other survey data collection activities were touted because of their speed and 
and low cost. 
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(see Sieber, Louis and Metzger, 1972). A large nurrber of federal proqram 
studies which include an extensive "design" phase, which includes field data 
collect lonr are implicitly based upon this traditional understandina of 
qualitative- data as "preliminary- and "hypothesis generating", while quantita- 
tive ddta IS "final" and "hypothesis testing". 

The Par allel Model 

The parallel model, on the other hand, makes no such assumption 
about an appropriate linear relationship between qualitative and quantitative 
method*-*. Typically such designs accept che arguments of Rist (1977) or 
Scriven (1972) that they represent two very different "ways of knowing", 
and tnat the researcher sees and illuminates different aspects of the social 
phenomenon under study. Thus, for example, one would not expect an ethnogra- 
phic treatment of a case to even touch upon the same patterns that the 
quantitative study is attempting to illuminate. 

This new tradition for multi-site, multi-method case studies has 
typically assumed that the most appropriate approach to maximizing the 
contributions of both is to allow them to develop independently, but simultan- 
eously. Thus, for example, in the study of the Rural Experimental Scnools 
Program, 10 ethnographers resided for three years at the local sites, while 
centrally located "quantitative" types desiqned surveys, multivariate data 
analysis plans, and so forth fHerriott, 1980). Neither had a deep influence 
upon the other, because there was little intellectual communication or 
exchange. The quantitative and qualitative components of this project were 
integrated, after a fashion, but this occurred post hoc , and the materials of 
each group were essentially "secondary data" for the other (see Rosenblum and 
Louis, 1981 for examples of how the ethnographic materials were used in 
quantitative analysis, and Firestone, 1980 for use of the quantitative data 
to enrich an ethnogr aphy ) - 

Parallelism reduced problems of coordination between competing 
-ways of knowing", and also eliminates many of the problems of coordination 
between centrally based staff and field based staff. In addition, it retains 
the maximum flexibility in design and analysis tot both methods. It is, 
thus, often preferred by paradigmatic researchers, and it is frequently 
employed m large studies (The Rand Study of Federal Proaram SuppDrtinc 
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FJducational Change, the SRI study of Teacher Corps, and the SDC study of 
Parent Involvement are some recent examples, in addition to the Abt Associates 
Rural Experimental Schools Study). However, it is not without its problems 
and costs. Fust, post hoc integration is extermely difficult, particularly 
where there is a great deal of qualitative and quantitative data. The 
process of integration is in most cases, similar to conducting a third study 
using secondary materials. Thus, integratibn across methods is usually 
limited, at best. Second, the parallel approach may reveal contraditions 
which, because of the autonomy of design, may be difficult to resolve 
(Trend, 1978). Finally, the approach can be very costly, since it requires 
tt.t two studies, one qualitative and one quantitative, be conducted simultan- 
eously, and resolved and integrated later.* 
The Fused Model 

Because of some of the limitations of the sequential and parallel 
approaches, th^re has been increasing emphasis in recent years upon an 
entirely different approach, which has gro^n up simultaneously in a number 
of research projects, but which .s perhaps best articulated by Yin (forthcom- 
ing and 1980); McClintock, et al , (1979); and Baltzell, (1980). The new method 
£uses some of the most "valuable" features of aaant itative data collection -- 
emphasis upon standardization of data {»ints, an emphasis upon determinina 
causality and testing hypotheses, rather than describing, and an emphasis 
upon cross-case analysis - with a flexibUe approach to observation and an 
emphasis upon holistic analysis. This approach is most frequently referred 
to as the ^tandardiz^Bd case method (see Baltzell, 1980). Overall, the key 
features of the approacii are: 

• considerable pre-specif icati on of the data that is to be 
collected in extensive protocols, which include not only 
questions to be answered, but specification of documents 
CO be collected and at least some of the respondants that 
must be interviewed or observed; 

• built in flexibility to pursue at least some additional 
topics and interview' different individuals from those 
pre-specified if these appear to be locally important 
to the social phenomenon under study; 



^I^oach being used in the ongoing SRI Teacher Corps Study, as 

described by Finnan (1980) may compensate, for it involved bringino field 
staf back Into the central office for 3 weeks of each month. However^ 
data collection and analysis apparently followed a typical parallel model. 
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• an emphasis upon a unit of analysis that is large r than_a 
respondent , e.g., a social unit, a classroom, a program 
as It IS implemented in a local context, etc.; 

• an tmphasis upon early data reduction and analysis while 
in the field, and by requiring a standardized repo rting 
format which involves pre-identifying at least the 
rna]or7ty of issues to be addressed; 

• the use of brief, but iterative approaches to data collec- 
tion. Typically there will be an initial round of field 
^TUTts, a period of analysis and refinement based on these, 
and a second round to collect data that is missing because 
of changes in the design and analysis plan , or to observe 
changes over time; (most studies using this approach spend 
between one and two weeks at a given site); 

• the development of causal arguments within each case and 
across cases using a direct replication design. The logic 
of this analytic approach is to find specific phenomenon in 
repeated cases under predictable conditions. Where the N 
IS very small (5 or less) the approach will typically 
emphasLze the internal validity of causal analysis within 
each case, where somewhat larger, the design often looks 
for systematic replications and attempts to establish the 
variability of conditions under which a phenomenon occurs. 

la addition, many of the standardized case studies have taken the approach 
one step further toward fusion: case studies, once they are completed, are 
coded by the field staff, using "survey" instruments similar to those that 
might be administered as a respondartt interview and analyzed quantitatively. 
(For a discussion of this as a technique, see Lucas, 1974; Yin, forthcoming; 
Prewitt and Eulau, 1975). Recent studies emphasizing this approach include 
Y.n's studies of Innovations in Urban Bureaucracies (1978) and of interorcani- 
zational networks between state, local and regional education aaencies 
(1981); King's study of staff development in desegr eaat ion ; and studies of 
Magnet Scnools as Mechanisms for Desegregation (Royster and Baltzell, 1979). 

Because the standardized case method does attempt to achieve a 
new synthesis of quantitative and qualitative paradigms, it has generated 
some controversy. Ethnographers, who are socialized to believe in the 
appropriateness of spending long periods of time in the field and allowina 
all hypotheses and interpretative frameworks to emerqe from field observations 
are appalled at the notion of standardising observations across sites, and of 
using reporting formats that are pre-determined and argue that the approach 
cannot possibly render an accurate portrait of the social system or settina. 
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Quantitative researchers, on the other hand, are concerned about the develop- 
ment of policy recommendations from small N research, using techniques that, 
to say the least, do not measure up to the rigor of typical sampling and 
measurement that tends to be preferred. In sum, the advocates contend that 
the approach maximizes the value of both paradigms, while opponents contend 
Ch2t It bowdlerizes the intent and value of both. 

Po licy Researcher as Sleuth; The Interactive Model 

Tfinal new approach to integrating qualitative and quantitative data - 
the interactive model -- builds ui»n some of the features of each of the 
previous three models. The ma^or distinctive characteristics of the approach 

ar e : 

• the merging of qualitative and quantitative data within as 
well as across sites; 

• staffing patterns which involve senior researchers who 
participate in both quantitative and qualitative aspects 
of the study; 

• persistant attempts to triangulate data sources and inter- 
pretations ; 

• cyclical interaction between the qualitative and quanti- 
tative m«thoH during ail phases of the study, including 
sampling, instrumentation, data collection, analysis, and 
report ing . 

Each of these features of the approach, as they were developed in the Study of 
the R&D Utilization Program, will be discussed briefly below. 

Pr ogram and Research Co n^= The R&D Utilization Proaram was an 
eight million "dollar demonstr^ion effort funded by the National Institute of 
Education between 1976 and 1979 to promote the adoption and implementation of 
new curriculum and staff development materials in 300 local schools. (For 
more details about the program, see IxDuis, et al , 1979. Ix^uis and Rosenblum, 
1980; and Louis, Rosenblum, and Molitor, forthcoming). Like many demonstra- 
tion efforts, there were delays in funding the research c mponent . NIE was, 
however, particularly anxious not to lose data on the early development of 
the program, and therefore funded a regional laboratory to perform some data 
collection during the first months of the program, and also encouraoed 
each of the seven demonstration projects to design relatively elaborata data 
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collection systems, including funding them to hire researchers to wr^e a 
total of 42x:ase studies about particular sites, whicn were site specific and 
relatively unstandardized. 

Abt Associates proposed to supplement the existing data collection 
activities with- Cwo waves of survey data collection, which would tap cross- 
site issues during early and later stages of implementatiion . The agency, 
however, expressed a preference for conducting a single survey, and asked 
the research staff to consider, alternative ways of providing longitudinal 
analysis of the process of change in schools. With NIE support the research 
staff developed a design which, while initially considered far from ideal, 
'finally produced a data base that allowed effective integration of qualita- 
tive and quantitative data across approximately 90 local sites involves in 

the program. , 

In the study of the R&D Utilization Program the emergenct interactive 
design was, in large measure, a byproduct of external features of the study 
context. At the time, many of these constraints were perceived as albatrosses 
by the research staff, , when initiating the study, had a marked preference 
for the sequential model described above. However, in the process of coping 
with external constraints, problems with availability of data, and data 
quality, a great deal was learned about how to maximize the utility of 
different approaches, "found" data, and both ad hoc and systematic information. 
In other wocds, t >e design that emerged is robust against many of the "normal 
crises" which occur in field based policy research. The specific elements of 
the design will be discussed below. 

Key Features of the Mechodoloaical Ap proach: First, the approach 
used in the study utilized all three of the models discussed above. The 
sequential approach was utilized, in that the single survey that was conducted 
in the portion of the study concerned with school change processes and 
outcomes was developed based on the qualitative data collection that preceeded 
It. The parallel model was used also, in that a standardized data collection 
operation completed by AAI staff, occurred simultaneously with the development 
of 42 -mini ethnographies" (Knapp, 1977) written by a field staff who had 
been previously hired by the funded projects and who were essentially indepen- 
dent of the emerging perspectives of the central research staff. Finally the 
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qualitative field data collection by AAI used the standardized case model 
for an additional 52 sites including the quantification of case materials. 

The integrated approach, however, is more than simply utilizing a 
variety of models for integrating qualitative and quantitative data across 
sites. Our approach introduced several new techniques for integration that 
are distinct from those discussed above. These included: 

(1) Merging Qualitative Data Sc irc^P... Within Sites: The Conso] idated 
Coding Process : Much of the discussion of integrating qualitative and quan- 
titative methods involves essentially cross-site analytic issues. The chal- 
lenge facing us included the cross-site merger of different data sources, 
but even more pressing was the problem of a diversity of within site data 
sources. One feature of policy and applied research is that it is frequently 
very "messy". For example, most major prograjus are not studied simply by one 
group of "evaluato-rs: or researchers. Rather, there may be internal evalua- 
tions, reportim and\dministrat ive data collection from the funding agency, 
and a sequence of external researchers. The multiplicity of lesearch and 
reporting requirements is a source of major concern to chose who are being 
studied, who^may frequently plaintively complain that "someone was here just 
last week asking me the same question". This was a particular problem in 
the RDU program, where "mini ethnographers", NIE administrative reporting re- 
quirements, two sets of external evaluators, and a variety of other research- 
ers interested in the program all descended with regulirilty upon the schools 
involved, but is increasingly a dilemma in other prograjus, such as TitLe I, 
IVC and others. 

For any, site, our data could include any or all of the following:* 

j 

j 9 demographic data; 

I • survey data of "key informants" collected at the-^-begin- 
\ ning of the program by the first evaluation contractor; 

• "event triggered" reports, discussing different phases 
of Che school's progress .through the project, which were 
management reports to NIE for monitoring purposes; 



"^Two other data sources were available at a later point in the study: 

• a survey of the principal; 

• a survey of between five ar^ ten teachers who were targeted 
users of the innovation adopted by the school. 
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• our ov.n field reports; which included a standardized case 
report and a quantified survey report filled out by the 
site visit team; 

• a -mini ethnography", and a "case study writers survey" 
which attempted to parallel some of the coding categories 
that were used by the AAI field staff; 

• various documents, provided either by the project, the 
local site, or other persons. 

In no more than 20 percent of the sites was a complete data base available, 
and in most cases major instruments or documents were unavailable. In sum, 
.we had a "missing data" problem of the first magnitude. Yet, it seemed 
fooljsh to throw away "evidence" of any sort, particularly given the high 
level of effort that had gone into collecting some of the data that was 
missing for the largest number of sites. 

Rather than analyzing each data source separately, it was decided 
to combine all data sources for each sit. in the "intensive" sample, e.g., a 
site where we had a "mini ethnography", or which had been site visited by an 
AAI staff member. This was done by developing a coding form, in which a 
single score or rating was given for each variable on the basis of a senior 
researcher's judgment derived from all the possible data sources enumerated 
above (with the exception of the formal principal a..d teacher surveys. 
The level of measurement was geared to maximizing the completeness of avail- 
able data. Thus, for example, the standardized case materials stressed 
precise measures of level of effort devoted to planning and implementation, 
in terms of staff days. The case studies tended to have much less precise 
measures. Level of effort was, therefore coded as "high-med lum-low" , where 
these were defined as ranges of staff days. Responding to these site based 
questionnaires which i^ludedJ40 dichotomous or Likert scale items was an 
extremely time-consum'- ng 'task, involving between two and three days of 
reading materials, and verifying responses on the cod-ng form. All coding 
was conducted by core senior staff members, who had made visits to at least 
four of the sites, and who went through an intensive two day session in which 
common interpretations of items were reached. 

While the process of "sleuthing" through the data trail -or each 
case did con. .e a great deal of time, what resulted was a quantitative data 
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base for 80 sites, which covered issues that could not be easily tapped 
through traditional survey methods, such as the "quality" of the decision- 
making proces-., patterns of influence of different actors over decisions, 
and so forth. Because of the diversity and large number of sites, we do 
not believe that any other form of data reduction and analysis would have 
produced a data base which at once reflected the "holistic" knowledge that 
site visit teams brought to the cases, but also the reliability of stand- 
ardized data, integrated both within and across sites.* And, while time- 
consuming, it was considerably less costly than other alternatives, such 
as (1) ignoring non-conforming data sources; or (2) gathering additional 
data on site in order to develop more complete standardized case studies. 

(2) Staffing ; The two cultures of research often result m research 
designs where staff members are specialized, either in qualitative data col- 
lection and analysis, or quantitative data collection and analysis. This ap- 
proach, while appropriate for the parallel model, does not facilitate inte- 
gration and interaction. The staffing patterns on the RDU study involved 
having the sair.e corf, staff of senior researchers involved in all instr^amen- 
tation, data collection and analysis. Thus, every individual who contributed 
as a major author or a-.alyst to the study .a' personally, familiar with site 
processes at all levels, and had responsih . or some portion of the quan- 

titative analysis in the study. This "integration by staffing" is, perhaps, 
one of the most effective ways of ensuring that the cycle of testing quanti- 
tative and qualitative observations occurs on a regular basis. 

(3) Trianqulation ^.-^d Data Quality ; As Webb, et al (1963) have 
pointed out, qualitative data becomes more compelling if observations are 
supported by multiple sources of evidence, or observations. The issue of 
reliability was of deep concern to us, in part because of the rather motley 
nature of our underlying data sources. Thus, our approach involved several 
approaches to triangulation: 

• inter-observer ; site visits were conducted in teams. 
Teams were required to reach consensus in preliminary 
rating of siu s on quantitative dimensions. In 
addition, when using the consolidated coding form, 
inter-rater reliabilities were conducted; 

*A limited inter-rater reliability check was conducted, which re- 
vealed an agreement rate of 72 percent. Several sites were eliminated be- 
cause of the amount of missing data. 
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• ho listic vs. ca t egorical ; as part of the interweaving 
of qualitative and quantitative data, findings from 

the qualitative data base were constantly tested out in 
the quantitative and vice versa. For example, in 
attempting to develop a categorical variable summarizing 
"site success", a definition of success was first dis- 
cussed among the staff on the basis of the field data 
visits. An indicator that reflected these discussions 
was developed from the survey data of principals and 
teachers. The "intensive" sites were then categorized, 
using the quantitative l^idlcator, and staff aaain 
discussed whether they were correctly classified, using 
the "holistic" judgements that they developed in the 
field visits, or f-'om reading the "mini ethnographies". 

• c ase vs. -surve y: in several cases, similar measures 
were built into teacher and principal surveys, and into 
the consolidated case coding instruments. Correlations 
between the perceptions of local respondents, and the 
perceptions of case raters were calculated, and several 
variables were discarded where correlations were not 
positive and significant.* 

• survey vs. survey : in all cases we were trying to ob- 
tain building level measures of the process and out- 
comes of implementing new curriculum and staff develop- 
ment materials. This meant aggregating teacher responses 
to the buildina level. In order to ensure that we were 
not falling into the "ecological fallacy", an analysis 

of variance between and within buildings, using both 
teacher and principal data, was performed. 
After the latter two activities were completed, robust 
measures were scaled to form a single school level score, 
reflecting the responses of principals, teachers and the 
external case material collected by field staff. 

(4) Cyclical Interaction Between Quantitative and Qua litative_ 
Data : In the I7quential and standardized case models, the assumption is that 
qualitative data will influence quantitative data collection and analysis, 
but noc vice versa . Most discussions of integrating quantitative and qualitat- 
ive methods also promote integration at o-.e phase of the execution of the 
study. Thus, for example, the sequential approach promotes integration 
during the period of instrumentation and final design for the "final" quanti- 
tative data collection; the parallel approach only after al] data collection 



*For a f~ther discqssion of case-survey tri angulation , see Ix^uis, 

Rosenblum, and Moll tor, forthcoinng ; Appendix B. 



and analysis in the separate streams is completed, and the standardized case 
approach during the process of cross-case analysis, during which counts and 
frequencies (or even correlations) are made. 

In the RDU study, on the other hand, quantified data was available 
from a very ear.y point in the study, both as a result of che surveys that 
were conducted prior to the award of the contract to AM, and also because 
the first round of field visits were coded while we were on site. The 
results of ear. descriptive analysis changed the focus oC cu. qualitative 
data collection in significant ways, for r.hey pointed out topics where there 
was little variation between sites (and cert^-in is ues that were expected to 
dominate t^.e case studies were therefore eliminated). In addition, they 
suggested some issues to look for -- such as the relationship of racial 
diversity to the process of change - that were not initially picked up on by 
the field visit or case study writ«>rs as significant. This cyclical inter- 
action between qualitative and quantitative occurred throughout the study, 
during staff analysis seminars and during the construction of instruments at 
various phases of the study. Cyclical integration stresses the 
following features: 

• ,nt»rAr-r ion bet ween qualitative a nd quantitative_in 
sampling: the interactive approach allows purposive 
sair. pl ing for cases to be combined with random samp_lin£ 
(or sampling of the universe) for survey or other 
structured data collection. TI-..-S feature maximizes 
discovery and generalizeabili ty . 

• interaction between qualitative and quant i tat ive__in 
ins trumentation : a constant interaction between 
Mutative and" quantitative analysis procedures 
produces an iterative approach to instrumentation, 
both for field data collection and more standardized 
coding instruments or data collection. Iterative 
instrumentation would be almost impossible for most 
standard longitudinal research design^, which involve 
repeated measures. This problem is avoided in the 
-consolidated coding- procedure, in which repeated 
measures are reconstr uctured from a bioad evidentiary 
base . 

• interactio n between qualitative a nd quant i tat ive in data 
^flection: the development of survey-type instruments 
for use even in early field data collection forced the 
research team to seriously think through measurement assump- 
tions at a stage where, in a mere typical sequential or 
parallel study, it would not occur. We believe that this 
contributed significantly to the quality of our measure- 
ment . 
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• interaction between qualitati ve and quantitative in 
liTi^lysis ; in the intetative model, it is impossible to 
"identify a distinct "analysis phase" m the research 
pto^ect. Rather, analysis begins with the first data 
collection event, and occurs periodically throughout the 
proDect. More importantly, however, is the fact that the 
simultaneous analysis of quantitative and qualitative 
data by the same staff continuously require a testing 

and verification of one data source against another, in- 
creasinq both reliability and validity. 

• interaction between qualitative and quantitative in 
reporting : while some of our reports are more "qualitative" 
"i^^ others more "quantitative" the immersion of all staff 

in both kinds of data hav2 meant that no report draws 
solely upon one data source. 

Some Reflection s ot^_ the Potential Stre ngths and Weakness es of the Interac_tive 
.Hodel for Soci - Policy Res earch 

Since there has only been one "test" of the approach that we have des- 
cribed above in a ma^or educational study, any assessments of its viability must 
be preliminary. However, it is appropriate to draw some conclusions about what 
has Deer, learned regarding the integration of quantitative and qualitative methods 
in design, daua collection and analysis. 

Design : Since the beginning of the "evaluation" movement, many disci- 
plinary researchers have complained about the lack of attention on the part of 
policy makers to designing "evaluable" programs, and their tendency to involve 
researchers too late, and at too low a level of effort to carry out the optimal 
research designs {Rodman and Kolodny, 1964, Weiss and Rein, 1969). Unfortunately, 
most of these please have not been attended to, for a variety of reasons wnich 
are often beyond the control of the research branches of various government 
agencies. Thus, as policy researchers, we are typically forced to choose 
between a research context that i3 less than ideal, or not conducting research 
on interesting and significant policy endeavors. The "sleuthing" strategy 
that characterizes the interactive model is premised on the assumption that 
the least promising circumstances can yield useable and even exciting data, 
that almost any piece of information can be turned into a "clue" to understand- 
ing the phenomenon in question, and that systematic analysis of clues is 
important . 
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r.. .n«r.«.v. approach ..ould not necessarily be v.ewea a. t.e 

,^ i^irlcQ the elegance or simplicity of a true 

'^"L::"rer;!: :.ea . ..e... .o^e . 
r::::^ .e.ea.. .... - 

elegant designs. Ainong these con<iitions are: 

. -^„^.-::,=ratir.r"^res!r.e^r.rr4rr/ 
. sir.;: "-.raraf^n^i ^:^^}:^^^ 

setting, and a broader cross-case analysis whicn 
dresses some of the same issues; 

• r;.s:.-,irj/a-rna;:rra^rVa'.ra^i:e 

"genetalize^le and reliiole" data; 
. ..ere the field reality is ".essy": lots o^^P-J^-f/ 
collected data exist but '^-\r']^'J^[lZ between sites, 
,anc. in quality and depth °f practical or 
but where response burden is ^^J^^^^'^'^^/u^.^ed to 
political issue; where .^^^^^ ^°°,here the 

^g.n afresh with a totally new f/^f^;,°^ j;^cess 
research involves documenting a longitudinal p 
that IS already well underway. 
..e.ae..e approach . no. a ..p.e one .e»e.., an. =a-.s Us 
.le„s It requires constant attention Uo. staff .e.be.s, co»».tt^, con 
problems. req ,„.Utat.ve and 

s.stent, .n-bouse senior staff, each ^ ^^^^ ^^^^ 

,„antrt.t.ve data collection and analysis. One ^ 

the need to maintain very low rates oi 
Sboald be emphasised is ^^Uection and analysis are 

prcect staff, since the p o ess o d s. . ^^^^ 

^"rracl at e pts trreplace .ey staff members d„ri. the 

:::: r.e rl .ere not at .U s.ccessf.l. ana It proved necessary 

of time in order to complete tne 

" r: Tn z:z ::z::zi:::^'^ — 

analyses. In addition, project 
.„st be relatively free of paradigmatic pre erence ^^^^^^^^ 
director or other key staff me,T>bers are reluct,.nt 

: :alitative and .antitative analysis, promoting cyclical integration 
may be extremely difficult. 

oat, collection. The -ide variety of different types of data collec- 
tion ,ct:::r::n;;::^^e camea o.t m the »U st.dy ano-s .s to address 
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»,.rd.„, vaUduy/.eU^.Uty t.at .ce often debated a^n, t.o.e w.o 
ate coneetned -u„ t.e .nctease .n „ult.-..te ,u.Utat.vely based studies. 

One of the major informal debates oocurrin, among qualitative cross- 
,lte researcners is the question of how one can best preserve the -holistic- 
insi,hts Obtained in a traditional ethn»,rarhy with the necessary truncating 
of time ih the field which occurs as financial resources are bein, stretched 
to cover many sites, ^e proponents of the -standardized case study- tend 
to prefer br.ef on-site data collection activities which are conducted 
exclusively by in-house staff members (e.g., a staffing approach that is more 
similar to traditional quantitative data collection procedures than to the 
true ethnographic approach,, ^e individuals who argue for -mini ethnograph- 
ies., on the other hand, contend that even a superficial understanding of the 
functioning of a pr»,ram. organization, or other social unit in a local 
setting must involve more time on site, more visits over a period of t me , 
addition to great-r flexibility in observations. Typically a -min, ethno- 
graphy- approach" argues for hiring trained individuals - often consultants - 
who are closer to the system in question, so that they can use whatever 
time IS available for observation and interviewing in a way *ich is more 
responsive to the unfolding of events at that site. 

Our study provided us with the opportunity to assess the richness 
and depth of data (validity, collected using a -mini ethnc^raphy- approach 
and the standardized case approach. 

Tl,. typical mxnx ethnography m our study consumed about 15-20 
.ays Of data collection and very preliminary wi.hin-site analysis on the part 
oe the case study writer, -o was, on average, a university professor who 

on the ,ob Of writing the case studies out of ^.sonal interest^ a 
.esult. there was, in a nun^er of c.ses , a gread deal of -contributed tim 

typical standardized case study, on the other hand, consumed approximately 
4-5 person days of data collection, which occurred at a single point in 

...T^priateness of the -Kg^ound of case s-dV -iters Jor 

conducting this type of -^"^-^J^" Tsed a caLlrdized case approach, 
of the 42 -mini ethnographies" actually usea a 
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time.* Two person teams were used, aud the team conducted some joint 
interviews with pr e-identif led key informants, and some individual interviews 
With teachers and ocner relevant respondents. Site cases were written by one 
team member on the basis of a team debriefing and discussion; each case was 
reviewed by the other team member and augmented where necessary. 

On tne whole, we believe that our standardized case studies yielded 
data that was equally i.nsightful into local site specific processes as 
the more intensive mini-ethnogr aphies . In addition, it was typically far 
more useful for the purposes of cross-site analysis. In a number of instances, 
we obtained both standardized cases and "mini ethnographies" on the same 
site. Where there were differences in what was emphasized in each case, 
m most instances our data were equally detailed, and equally informative 
about the longitudinal processes of change. In no case where we were familiar 
with the mini-ethnography sites, did we find any major contrad.tctions 
between the data that we had collected, and that collected and reported 
through more intensive, less standardized means. 

Two conclusions may be drawn from this informal comparison, which 
was corroborated not only by our possibly biased project staff, but also by 
several individuals who served as consultants to the study who were familiar 
with both types of quantitative case data. First, the more costly "mini 
ethnography" does not result in sufficiently better data to justify the 
additional use of resources. Second, the standardized case study can 
reveal local site uniqueness {if the reporting format allows this) and, 
in addition, is far more useful for cross-site synthesis. Mini-ethnogr aphies 
are no better at reflecting "holistic" patterns within the site because 

*It should be emphasized that the 4-5 days per site that yielded 
relatively valid data for our study should not be generalized to other 
settings. For example, when examining district-wide behavior, it is 
clearly necessary to have a somewhat larger number of person days than 
when examining program functioning in a single school. However, the 
general principal of contrasting mini-ethnographies and standardized 
case materials still holds. 

Further, it should also be emphasized that information gathered 
during the 4-5 days on site was augmented by review of documents prior 
to and after the visit, and through familiarity with the nature and 
substance of the intervention gained in interviews with managers at the 
funded project level and with the external providers of technical 
assistance . 
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^..1 feature of a true ethnography: a continuous, 
they stai lack the essential feature observation m 

on-site presence which involves substantial participant 

addition to interviewing. concerned 
validity IS not the only problem, however. Vr.ose 

3„.. ^ -the .n^luenc. c£ pc.nc.pal eve t.e ae ^^^^ 

- - tr;r::rjrr::::r.::.:s.^. ...^ 

e..„ea ..c. ,u, t e ^^^^^^^^ ^^^^^ 

*.t sh.res „ se..stru«u.e .tructuce a,t. coUec- 

^~ acfeve .... levels of „.ee.e„t .e.ween auee.ent 

tion techniques and still acniev 

sources of data. ^r^nrrpte measures of 

.et us ta.., fo. example, one of our less — « ^ ^^^^^ 
.vel ou.o„es, -ove.ll o...».o„.l ^^^^ ^^^^^ ^^^^ 

e.s. .ou„a Of p---;;;;; " '^^^^^ ,„a .„p.ove.e„. 

there were many spin-off effects ^ „„n.,tion of a new 

, ,11 cases, to the implementation or 

that were not directly tied, nK^rale, improvement m 

»se included changes m staff morale, f 
curriculum package. -s*^ developed a 

of the school in the community, and the -ike. 
the image of the schoo ^„,fits that we had observed, and 

set Of Likert scales reflecting side benefits th ^^^^^^^ 
during our ma.or qualitative data collection ^ ^^^^^^^ 

^h,^ battery. The battery was also inciuae 
.ere visited on thi b tt y ^^^^^ ^^^^^ ^^^^^ ^^^^ ^^,,,,3 

teachers and principals. While ^.dants on broad school organiza- 

3omewhat leS3 generously than indigenous ----^ ^^^^^ 

.onal impacts, the- ratings ^ J^.l^.^ . These are quite 

.55 with the principal ratings, and .44 ^^^^ 

topical Of the intercorrelations between data sources that 

xtems or indices which overlapped. 
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we we.e, finally a.Ie .o achieve even ..eacer .eUa..Ucy in our 
. "a a set by co.b.n.n. school scores on our outcome variables 
quantitative data set y "consolidated coding 

f.om both originally qualitative sources (e.g., 

form") and from survey data.* 

IS that this analysis visit, 
o K .nalvsis IS an incremental feature associated with 
Rather, analysis is on K»fnrP the design of 

* f iH Hata collection activities, before tne ae y 

research s--a££ , 9^ P ^^^^ ^^^^ 

aevelop ..U^le. r^.Utat.ve and ,ua„tita»ve occu..ed. 

Evacuee abound w..ch tha .«.,ra»o„ of ,„^,„, 

Eac. .... a f.eld sue «a. ..s faced ^^^^^^^^ ^^^^ ^^^^^^^ 

abouc the local site processes on a 1,1 

the phenomena 

--^"r r :: ::::rir:r: u a.ffe.e« .spo„ae„t.s 

that had been observed, the ^^^^^ 

i iQSUP in questions, and whetner 

te.-spectives on the issue in , ^^^^^ ^^^^^^ 

features of what was happening at the 

ee.., instruments as they existed, ^is * , ^^^^ 

aeld and dutin, extensive debtiefin, sessions with enti 

H that (11 we wete constantly pressing to have our 
s.aff of eight ensured noUstic 
quantitative instrutents reflect, ouc Po ^^^^^^^ 
understanding of how sites were operating, (2) we 

nrprise understanding ot tne si'-c 
through quantification, a more precise ^^^^^^^ 

1 r.f HPtail in each case write up, 
reflected in the level of detail standardization 

rr:r;. ■•;::::~rr;.:.;: . ....... 

of a very large number of sites. 

* hho -nualitv" of the quantitative 

JF^Tl^e extensive f f/, ,'p;:ndix B, in Louis, Rosenbl 

data that was derived from various sources, PPe 
and Molitor, forthcoming. 
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. Ko =aid to be either quantitative 

,.,U.«.v,. rc. ,...ple, can a a . ^ ^^^^^^^^^^^ ^^^^ 

.„..r.ly dependent on the ..ec.twe, hoU t.= : ^ ^^^^^^^ 

onlv quantitative? While tne 
f.eld teams be described as only q ^^^^^^ ^^^^ ^^^^^^^ .nterpre- 

used to manipulate the data are statistica , ^^^^ ^^^^^ 

^^tAllv conditioned by its origins, 
of .e.ults, tot. c ^^^^^^ ^^^^ 

.e approach any ,.ve„ analyses us ng ^^^^ 

— - - :r:t a co.e.at.o„a. .e.at.o„s.ps that 
Kno-ledge o£ the descriptive statistics ^„,,,i„„ had endej. 

— orti: ::i:r:::r:r rrrL, a. t. r^. 

- Trstrro- ::ar;ratr:;::Uita- 
::::::: o. rust. .a„^^^ ^^j^^^-- 
- — ^^rordirire r„s, ..est.... t..., 

^33,,l, even cc.putin, ra„K ^^^^^ ^ ^^^^^^^ 

" ^ r^nrtr:; ::^TJe :: - -^^^^^ 

,5 - the abil y ^^^^ 

— cases di^inishe - ^^^^ ^.^^ .est or .est 
typical one, perhaps, is ^^^^ 

.»iuar data informants ---J- „^^,,,,„, .^se in policy 

cation .ey variables - ^ • 

research, the n„-,e ^ „„,„estiona.Iy in a condition where Eor.aU.- 

„u»*ers in the hundreds, ve ar q <,„,„t lEication , 

.on o. data analysis pr.ed. s _e. ^^^^^^^^ ^ ^^^^^^ ^^^^^ 
Whether highly rigorous or less so 

data. nroi^ct, it IS highly recommended 

on the »p«riences o ^ ^ ^ ^„ „ effect- 

that such .antification occur e«l, a-^iHll— I ^^^^^^^^ 
.ely integrate the process '^^^ ^^^^^^^ „„,,,.s and 

^-^acfi ;?nalvsis which IS neces5<ai.i-».jr 
^e cross-cas analy ^^^^ ,3 

c.oss sections of a larger P „„,„,i„tive and qualitative data 

fact that the simultaneous analysis q ^^^^j^„,,„„ of one 

the same staff continuously require a testing and 
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report, but which does Iittie 
analysis as a legitimate methdology. 
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